Data switching networks are used to route data items between devices, for example between servers in data centres. Ethernet is a frame-based networking technique primarily used for local area networks (LANs). A typical Ethernet switching network includes one or more Ethernet bridges that house switching elements for routing data between a plurality of data ingress ports and a plurality of data egress ports.
A Link Aggregation Group (LAG) is a collection of physical network links brought together to provide a single logical channel of higher bandwidth. The Link Aggregation Control Protocol (LACP) is part of the IEEE specification 802.3ad. FIG. 1 of the accompanying drawings shows a typical Ethernet LAG. First and second Ethernet bridges 2 and 4 are connected by Ethernet links 10. The first bridge 2 has a plurality of data ingress/egress ports 6, and the second bridge 4 has a plurality of data ingress/egress ports 8. In this simplified example, the LAG has four links 10 connecting the two Ethernet bridges 2 and 4 providing up to 4 times the bandwidth of a single link. The LACP allows a network device connected to an ingress/egress port of a bridge to negotiate an automatic bundling of links by sending LACP management frames to the peer. LACP works by sending frames (LACPDUs) down all links that have the protocol enabled. If a device is present on the other end of the link that also has LACP enabled, it will also independently send frames along the same links enabling the two units to detect multiple links between themselves and then combine them into a single logical link. LACP can be configured in one of two modes: Active or Passive. In Active mode it will always send frames along the configured links. In passive mode however, it acts as “speak when spoken to”, and therefore can be used as a way of controlling accidental loops (as long as the other device is in active mode). Link aggregation presents a number of problems for Ethernet-based communications. Ethernet networks use a defined protocol to determine the spanning tree of the network and disable any links that offer alternate paths to a destination. This ensures there is only one route from a source to a destination. Additionally, there is only one source to destination connection for network traffic that uses an Ethernet LAG. This means that a particular source to destination connection will always use the same link within the LAG. A LAG uses a distribution algorithm based upon source and/or destination network addresses of the connected devices to define the connections to be made on the individual Ethernet links that form the LAG. Typically a number of different distribution algorithms are available to a network administrator. The network administrator will choose the algorithm that gives the most even distribution of network traffic across the LAG depending upon a number of pre-determined, site specific parameters.
The distribution algorithm is normally based on either the source or destination MAC addresses or a combination of both the source and the destination MAC addresses to select an Ethernet link within the LAG for the Ethernet frame to be delivered on. Other distribution algorithms may use the IP address and port numbers of the layer 3 protocol.
Having a single route or path for the packets to follow does simplify a number of issues. Ethernet packets should be delivered in order. Having a single route from source to destination guarantees that one packet cannot overtake another. This greatly improves the performance of higher-level protocols. Ethernet is normally the data link layer for TCP/IP and UDP/IP. TCP can receive IP packets in a different order from the order they were sent but this requires a significant processing and buffering overhead at the receiving node.
However, the simplification of a single route also means that the packet load, over the LAG, can be very unbalanced. Full utilization of the LAG requires all links to be transmitting data. This in turn requires a number of communications to occur concurrently with routes that send the data along different links of the LAG. The probability of better utilization increases with the number of different conversations being relayed across the LAG. The greater the number of links in the LAG then the greater the number of conversations are needed to keep all the links of the LAG active.
A distribution algorithm is required to implement a LAG. This will select the link that is used to relay a packet across the LAG. This has to be executed at the same time that the MAC tables are accessed to perform the MAC address to destination port translation. This usually restricts the width of the LAG to a proportion (no more than half) of the links on the Ethernet switch chip. The physical implementation of an Ethernet switch usually puts an upper limit on this number. FIG. 2 of the accompanying drawings gives an example of a multi-switch LAG. Having first and second Ethernet bridges 12 and 14, each of which has two layers of internal switching, provided by respective pluralities of switches 16 and 18, interconnected by internal links 20 and 22 respectively. Each bridge 12, 14 has a plurality of ingress/egress ports 24, 26 through which data passes for routing by the switches 16 and 18. The bridges 12, 14 are connected to one another by a plurality of Ethernet links 28 over which data is passed. These Ethernet links are arranged into a LAG 30, in order to provide an effective increase in bandwidth between the bridges 12 and 14.
The example shows each Ethernet bridge having two layers of switching. However, the method works equally well with many more layers of switching and also with switch chips with many more ports per switch than the 8 shown per switch in the Figure. The internal topology used in FIG. 2 is also an example and the method works equally well with other topologies.
In the example of FIG. 2, all of the Ethernet links 28 of the switching elements 16, 18 next to the LAG are used within the LAG. This is a configuration that could be made to work but it is restrictive to a network administrator to impose this structure for a number of reasons:
1. The physical locations of the cables may not conveniently reach these ports or a non LAG Ethernet link may be required from one or more of the ports located in the natural range of the LAG ports. 10 Gbps Ethernet copper cables have a relatively short reach and this can impose additional physical constraints.
2. The requirements of the LAG may change over time and additional ports may need to be added or removed.
3. Individual links of the LAG may fail and additional links may need to be added to cover the extra load or the distribution of traffic on the other links may need to be changed to cover the traffic that was using the failing link.
FIG. 3 of the accompanying drawings gives an example of a multi-switch LAG which is similar to the example shown in FIG. 2. In the FIG. 3 example, however, some of the links 34 within the range of LAG 30 are assigned as normal Ethernet bridge links outside of the LAG 30.
Ethernet LAGs are typically limited to a small number of Ethernet Links, perhaps 4 or 8. At the time of writing the maximum available 10 GbE links in a LAG was 16.